Supplement of “ Limited-memory Common-directions Method for Distributed Optimization and its Application on Empirical Risk Minimization”

II More Experiments We present more experimental results that are not included in the main paper in this section. We consider the same experiment environment, and the same problem being solved. We present the results using different values of C to see the relative efficiency when the problems become more difficult or easier. The result of C = 10−3 is shown in Figure (I), and the result of C = 1000 is shown in Figure (II). For C = 10−3, the problems are easier to solve. We observe that L-CommDir-BFGS is faster than existing methods on all data sets, and L-CommDir-Step outperforms state of the art on all data sets but url, but is still competitive on url. For C = 1000, L-CommDir-BFGS is the fastest on most data sets, and the only exception is KDD2010-b, on which L-CommDir-BFGS is slightly slower than L-CommDirStep, but faster than other methods. On the other hand, L-CommDir-Step is slower than TRON on webspam but faster than existing methods on other data sets. These results show that our method is highly efficient by using (2.9) or (2.10) to decide Pk. The other choice, L-CommDir-Grad, is obviously inferior for most cases. We also modify from TRON to obtain a line-search truncated Newton solver to compare with our method. This line-search truncated Newton method is denoted by NEWTON in the results in Figures (III)-(V). Results show that NEWTON is consistently the fastest on criteo for all choices of C, and outperforms L-CommDir-Step in the later stage on url for all C, and on epsilon for C = 1000, but LCommDir-BFGS and L-CommDir-Step are faster on all other cases. Therefore, in most cases, our method is the most efficient one.